TODO:

• port feature: user-defined differentiable function
  • In Python, it provides the class Function. I also curious about how this could be implemented in Julia. If we defined a abstract type like CustomFunc, how can we make it callable?
• test cases:
  • attach_grad!
  • getgrad
  • mark_variables!
  • symbol
  • record
  • pause
  • train_mode
  • predict_mode
  • backward!
• leverage grad_req_map from update OpReqType enum #283
• update NEWS.md
• docs
• figure out this: ndarray: porting Python's autograd #274 (comment)

We don't have custom operators yet so I don't thing we need user-defined differential functions, (it is actually a very hard problem to solve in Julia...)

In Python, this shows that we only need one c api: MXCustomFunctionRecord.
So.. IMO we still can have user-defined diff funcs.
Also, IIUC, make_symbol translates the style of NDArray (dynamic) to Sybolic (static). Maybe AutoGrad can reveal a another way to create custom layer via combining make_symbol and user funcs?

About implementation of user funcs, maybe meta programming can help, just my imagination:

@mx.registerfunc MyFunc  # create callable function for it
forward(f::Myfunc, ...) = ...
backward(f::Myfunc, ...) = ...

but not sure where is the pitfall of implementing it.

found a problematic case:

julia> x = mx.NDArray([1 2; 3 4])
       mx.attach_grad(x)
       
       y = mx.record() do
         1x .* x
       end
       
       mx.backward(y)

julia> copy(mx.grad(x))
2×2 Array{Int64,2}:
 1   4
 9  16

It's gradient shoud be same as the one in commit message.
something wrong in broadcast?

I would leave custom functions completely out of the picture for now. The problems and pitfalls are plenty and I am currently not convinced there is a good way to implement them (outside of #173, which is currently stalled).

In order to make customs function work you need to guarantee that they are never called asynchronously.

In order to make customs function work you need to guarantee that they are never called asynchronously.

which doc of mxnet c api can I consult? still can not understande what is the issue of aynchronous (io?).

See the discussion and linked abandoned PRs in #166

Thanks! About custom OP, I think the brief summary is that libmxnet has a async engine when executing operators, so the Julia callback will be called in a different thread than the main thread. Due to the current incomplete support for multi-thread GC in Julia, a callback being called in a different thread from the C++ side is not guaranteed to function properly.

Codecov Report

Merging #274 into master will increase coverage by 1.2%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #274     +/-   ##
=========================================
+ Coverage   69.74%   70.94%   +1.2%     
=========================================
  Files          25       26      +1     
  Lines        1963     2034     +71     
=========================================
+ Hits         1369     1443     +74     
+ Misses        594      591      -3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db09528...0b11e4a. Read the comment docs.

For record, here is an API changes:

• Second order gradient and Subgraph execution apache/mxnet#7698

https://github.com/apache/incubator-mxnet/blob/7484437d21c5e1456c24bff31e6642cf0a47db4d/src/imperative/imperative.cc#L182-L197

I ran into this error with recent libmxnet master,
seems that we should get rid of any use of op(...; out = ...)

I think this PR is ready for review.
I will send another PR for doc and docstring.

I try to implement user-defined function.
will send a PR after this PR merged.
https://github.com/dmlc/MXNet.jl/tree/ib/nd-ad-customfunc

@pluskid need more time to review?

travis failure is macOS build timeout.